release: GeoBrix 0.5.0 — virtual tiles, v2 tile struct, CRS families - #72
Draft
mjohns-databricks wants to merge 828 commits into
Draft
release: GeoBrix 0.5.0 — virtual tiles, v2 tile struct, CRS families#72mjohns-databricks wants to merge 828 commits into
mjohns-databricks wants to merge 828 commits into
Conversation
mjohns-databricks
temporarily deployed
to
runtime
August 13, 2026 14:35 — with
GitHub Actions
Inactive
mjohns-databricks
temporarily deployed
to
runtime
August 13, 2026 14:35 — with
GitHub Actions
Inactive
mjohns-databricks
temporarily deployed
to
runtime
August 13, 2026 14:35 — with
GitHub Actions
Inactive
mjohns-databricks
had a problem deploying
to
runtime
August 13, 2026 15:02 — with
GitHub Actions
Failure
mjohns-databricks
temporarily deployed
to
runtime
August 13, 2026 20:45 — with
GitHub Actions
Inactive
mjohns-databricks
temporarily deployed
to
runtime
August 13, 2026 20:46 — with
GitHub Actions
Inactive
mjohns-databricks
temporarily deployed
to
runtime
August 13, 2026 20:46 — with
GitHub Actions
Inactive
mjohns-databricks
temporarily deployed
to
runtime
August 13, 2026 20:46 — with
GitHub Actions
Inactive
mjohns-databricks
temporarily deployed
to
runtime
August 13, 2026 21:12 — with
GitHub Actions
Inactive
mjohns-databricks
had a problem deploying
to
runtime
August 13, 2026 21:21 — with
GitHub Actions
Failure
mjohns-databricks
added a commit
that referenced
this pull request
Aug 13, 2026
PR #72's build failed on `black --check`. CI lints as `isort && black && flake8 src test`, so black's failure short-circuited before flake8 — hiding pre-existing flake8 debt that surfaces once black passes. This clears both in one pass: - black-reformat 7 drifted files (bench/{runner,spec}.py, test/bench/{cluster, results,spec}, test/pyrx/{core_agg,v2_tile_output_invariant}). - flake8: drop 3 unused imports (F401 in pyrx/_serde.py, test/rasterx/ test_udtf_error_row.py, test/vectorx/test_crs.py) and mark the bench helper run_spark_path with # noqa: C901 (matches the _render precedent). No functional change; none of these files were part of the vizx/eo-series work — this is latent tree-wide lint debt that was masked by the black short-circuit. Co-authored-by: Isaac
mjohns-databricks
temporarily deployed
to
runtime
August 13, 2026 22:26 — with
GitHub Actions
Inactive
mjohns-databricks
temporarily deployed
to
runtime
August 13, 2026 22:26 — with
GitHub Actions
Inactive
mjohns-databricks
temporarily deployed
to
runtime
August 13, 2026 22:26 — with
GitHub Actions
Inactive
mjohns-databricks
temporarily deployed
to
runtime
August 13, 2026 22:26 — with
GitHub Actions
Inactive
mjohns-databricks
temporarily deployed
to
runtime
August 13, 2026 22:54 — with
GitHub Actions
Inactive
mjohns-databricks
temporarily deployed
to
runtime
August 13, 2026 22:55 — with
GitHub Actions
Inactive
mjohns-databricks
had a problem deploying
to
runtime
August 15, 2026 22:05 — with
GitHub Actions
Failure
mjohns-databricks
temporarily deployed
to
runtime
August 15, 2026 22:05 — with
GitHub Actions
Inactive
mjohns-databricks
had a problem deploying
to
runtime
August 15, 2026 22:05 — with
GitHub Actions
Failure
mjohns-databricks
temporarily deployed
to
runtime
August 15, 2026 22:05 — with
GitHub Actions
Inactive
…nvention Two additions before authoring, per user emphasis that VectorX is geometry-dominated: (1) a uniform output-representation convention (elision ... + format annotation for [E]WKB/MVT binary, [E]WKT, GeoJSON, CRS strings; ~60-char cell cap) stated once in the T1 Conventions section and enforced by the docs-examples guard; (2) an explicit per-function review gate that all four tabs use the SAME example (same fixture/op/args) so output renders consistently, with genuine tier differences labeled, never fabricated or hidden. Co-authored-by: Isaac
…extension Adds the VectorX tabbed-docs foundation layer that T2-T5 build on: - Extend generate-function-info.py _TIER_SCANS: widen light-Python glob from rasterx_*_python_light.py to *_python_light.py (picks up the new vectorx light file), and add vectorx_functions.py as a second python-heavy scan path. RasterX bindings unchanged (spot-checked rst_avg/rst_width). - Add VectorX fixture loaders to _fixtures.py: tin_df/tin_df_heavy (4 WKB POINT Z, non-flat Z range 10 m), mvt_features_df/..._heavy (2 tile- local WKB POINTs), geom_ewkt_df/..._heavy (SRID=4326;POINT(13 42)), and legacy_geom_df/..._heavy (Mosaic InternalGeometry struct). Adds setup view builders: create_setup_views_vectorx_light/heavy for the four views (tin_survey, mvt_features, vector_geoms, legacy_geoms). - Add 8 new fixture tests in test_fixtures_helpers.py asserting non-degenerate data (rows>0, WKB size, Z range, EWKT has SRID=, struct typeId=1, all 4 views accessible after setup). - Create vectorx_functions_python_light.py scaffold: module docstring, fixture helpers, setup example; per-function examples deferred to T2-T5. - Create test_vectorx_functions_python_light.py: 4 smoke tests (import, setup exists, setup executes, views created). Autouse fixture registers pyvx + creates the four VectorX views. - Regenerate function-info.json via gbx:docs:function-info; 180 entries, VectorX bindings remain ["sql"] (light/heavy examples not yet authored). - Add Setup + Conventions section to vectorx-functions.mdx (after tier- availability table, before Vector tile output): placeholder->view->backs table, 4-tab reading guide, uniform output-representation convention ([E]WKB/MVT/WKT/CRS ~60-char cap), CRS-family-returns-BINARY note, light-UDTF LATERAL form convention. Import FunctionExamples component + vectorxLightCode + scalaApiExamplesCode for T2-T5 use. Test: 1100 passed, 1 deselected, 0 failed (baseline 1088+12 new). Docs build: gbx:docs:build --no-restart ✓. Co-authored-by: Isaac
…t_transformcrs) Adds 4-tab (SQL / Python light / Python heavy / Scala) examples for the three CRS-family VectorX functions and wires them into FunctionExamples blocks on the vectorx-functions doc page. Changes: - vectorx_functions_python_light.py: 3 light examples + outputs (pyvx tier, uses vector_geoms setup view, POINT(13,42) SRID=4326 fixture) - vectorx_functions.py: 3 heavy examples + outputs (vectorx tier, same fixture) - ScalaApiExamples.scala: 3 Scala examples + outputs - test_vectorx_functions_python_light.py: 3 per-function light tests asserting real values (EPSG:4326 string, non-null EWKB bytes) - test_vectorx_functions.py: vectorx_heavy_setup fixture + 3 heavy tests - vectorx-functions.mdx: CodeFromTest → FunctionExamples for st_crs / st_setcrs / st_transformcrs (bindings=[sql, python-light, python-heavy, scala]) - test_example_output_tables.py: extend _example_modules() to scan vectorx files - function-info.json: regenerated; all 3 CRS functions now carry all 4 bindings Co-authored-by: Isaac
C1: st_setcrs and st_transformcrs SQL tabs wrapped the function in gbx_st_crs(...), showing STRING instead of the BINARY the functions actually return. Rewritten to direct invocations showing the real BINARY output with (EWKB binary — ...) annotation. C2: All 3 CRS SQL examples used inline literals and multi-column showcases differing from the Python/Scala tabs (POINT(11,42) vs 13,42; multi-col vs single-col). Aligned to the shared vector_geoms fixture (SRID=4326;POINT (13 42)) and single-column form matching all other tabs. Also: added test_st_setcrs_stamps_different_crs (SQL + light) that stamps EPSG:3857 on an SRID=4326 geom and round-trips through st_crs to verify the SRID actually changed — prev test stamped 4326 onto 4326, a silent no-op. Updated vectorx_registered fixture to create vector_geoms view so SQL examples execute. Regenerated function-info.json. Co-authored-by: Isaac
…ramid)
Add 4-tab FunctionExamples blocks for both MVT functions, converting from the
prior single-tier CodeFromTest to the full SQL/Python-light/Python-heavy/Scala
standard. Fixtures reuse the `mvt_features` view (tile-local WKB POINTs) for
st_asmvt and inline WGS-84 POINT(0,0) for st_asmvt_pyramid.
Fix a correctness bug in the stalled implementation: the heavy Python and Scala
examples incorrectly referenced `t.tile.z` / `t.tile.mvt_bytes` for the
DataFrame API struct path. The generator `.alias("t")` yields direct fields
`t.z`, `t.x`, `t.y`, `t.mvt_bytes` (no `tile` wrapper); the `tile` wrapper
only appears in SQL `LATERAL VIEW ... AS tile` syntax. Switched to
`.selectExpr("t.z AS z", ...)` and `rows[0]["mvt_bytes"]` to match the
confirmed unit-test pattern in `test_st_asmvt_pyramid.py`.
Also add the missing `st_asmvt_sql_example_output` and
`st_asmvt_pyramid_sql_example_output` constants (eliminating build warnings)
and regenerate function-info.json so both functions gain all 4 tier bindings.
Doc tests: 1126 passed, 0 failed (baseline 1088; +38 from T1–T3).
Docs guard: OK (431 output constants, 511 tabs, 124 annotation groups).
Build: clean (0 warnings).
Co-authored-by: Isaac
…expression Chaining Column-API tile functions in one expression (rx.rst_slope(rx.rst_setcrs(...))) fuses into a single Python worker, so intermediate tiles pass in-worker without a JVM round-trip — the per-batch boundary is crossed once for the whole chain, not once per step. Materializing intermediates across separate select/withColumn steps re-serializes at each stage. Composes with virtual-tile deferral. (Insight surfaced from the FILE Python-UDF execution design; the note is about current chaining behavior, not FILE.) Co-authored-by: Isaac
…cation Three correctness fixes on the T3 MVT tabbed examples: 1. SQL examples returned `length(gbx_st_asmvt(...))` (INTEGER) instead of the raw BINARY, violating "the shown code invokes THE NAMED FUNCTION and nothing wraps it to change the result type". Dropped the length() wrapper; both SQL examples now return the BINARY column directly (matching the `... (MVT binary)` annotation). SQL tests updated to assert non-null/non-empty bytes instead of a byte-count integer. 2. `st_asmvt_pyramid_python_light_example` returned `rows[0]["z"]` (an int), leaving the MVT encoding path unverified on the light tier. Changed to return `rows[0]["mvt_bytes"]`; test updated to assert `isinstance(result,(bytes, bytearray)) and len(result) > 0`, mirroring the heavy tier test. 3. Pyramid output annotation was split across tiers (SQL/light used "one row per intersecting tile across zoom levels 0–2"; heavy/Scala used "one row per tile; use t.z …"). The annotation consistency guard requires identical trailing annotations on identical tables; unified all four tabs to the SQL/light phrasing. Scala minor: `result.count()` → `result.show()` so the shown code matches the displayed table output. Doc tests: 1128 passed, 0 failed. Guard: OK (126 annotation groups). Build: clean. Co-authored-by: Isaac
…n scoping Doc-tests under api/ can only run as the whole dir (the _fixtures sibling import needs whole-dir collection), so there was no way to run just the changed functions. Add --filter '<expr>' which passes pytest -k while still collecting the whole path, so tasks touching a few functions run only those tests (e.g. --path api/ --filter 'st_triangulate or st_interpolate') instead of the full ~1128-test api suite each iteration. The host-only gbx:test:docs-examples guard remains the cross-tab consistency net. Co-authored-by: Isaac
Adds per-function SQL/Python-light/Python-heavy/Scala examples for
st_triangulate, st_interpolateelevationbbox, st_interpolateelevationgeom.
Converts three CodeFromTest blocks in vectorx-functions.mdx to
FunctionExamples and removes the now-redundant hand-written PySpark block.
Key implementation findings surfaced during authoring:
- TIN Scala expressions extend CollectionGenerator, not UDTF. SQL tab uses
LATERAL VIEW (Hive generator syntax); light Python tab uses SQL standard
LATERAL (pyvx UDTF). This matches the st_asmvt_pyramid pattern already
in the docs.
- Single-field elementSchema (STRUCT<triangle BINARY>) unwraps the field
type directly in the Python Column API: .alias("triangle") not
.alias("t").select("t.triangle"), which fails with INVALID_EXTRACT_BASE.
- Spark SQL decimal literals (0.01) fail readDouble in the Scala expressions;
integer literals (0) are used for tolerances (accepted as Int→Double; same
behavior on the clean 4-point fixture).
All 3 functions now carry all 4 bindings in function-info.json.
Full api/ doc-test suite: 1143 passed (+15 vs baseline), 0 failed.
Co-authored-by: Isaac
Remove the outdated "_should_stream docstring" note that called OpenFileGDB memory-bounding "a separate follow-up". The bounding was already in place via _write_local_osgeo_gdb, which streams fragments one at a time with OGR transaction batching. Update the docstring to accurately describe the current implementation. Add test_vector_gdb_streaming.py with two tests: - test_write_local_osgeo_gdb_reads_one_fragment_at_a_time: unit-level spy via WeakRef-tracked feather.read_table calls, asserts max concurrent live fragment tables <= 1 (GAP-4 streaming invariant). - test_gdb_multi_fragment_roundtrip_rows_geoms_crs: end-to-end multi-partition FileGDB write+read verifying row count, attribute values, geometry types, and CRS round-trip fidelity. Co-authored-by: Isaac
…self-report Add a static regression guard (test_materialize_regression_guard.py) that asserts each of the four materializing entry points (file_ref_arg, RasterGbxWriter.write, _fromfile_impl, CogGbxWriter.write) still references the shared size-safety gate (_connect_aware_lru_sizing / materialize_decision). A future refactor that drops the gate call and substitutes a raw unbounded .read() will fail this test immediately, before reaching Serverless. Add report_detected_cap(spark=None) -> int to ds/file_gbx.py (__all__) so the T8 on-cluster validation can assert that the Serverless Connect session detects the 64 MiB cap (the linchpin — a mis-detect silently applies the 256 MiB classic cap, which is the OOM vector on Serverless). Add 5 unit tests for report_detected_cap in test_materialize_decision.py, including a consistency check proving report_detected_cap and materialize_decision use the exact same threshold boundary. Add one CLAUDE.md bullet in the GDAL resource management section codifying the Serverless-safe materialize policy as a REQUIRED project convention. Co-authored-by: Isaac
…ize behavior Add serverless-and-memory.mdx as the authoritative reference for connect-aware memory caps (64 MiB Serverless/Connect, 256 MiB classic, GBX_STREAM_MAX_BYTES), materialize-vs-virtual safety, FILE Delta-table fast path (~1.8–2 s / ~16–17× faster than directory scan on Serverless), memory-safe write/ingest guarantees, and explicit caveats for rst_fromfile(materialize=True), rst_fromcontent, and large-tile write with pending warp/clip. Correct the reader Serverless matrix: update directory scan timing from ~74 s to ~30 s / 1k tiles; update the ~41× speedup claim to ~16–17×; reframe the FUSE-of-FILE path as a memory-safety mechanism (not a speedup) in the Read resolver section; add connect-aware cap documentation throughout. Add :::caution admonitions to rst_fromfile(materialize=True) and rst_fromcontent in raster-functions.mdx. Wire the new page into sidebars.js. Co-authored-by: Isaac
The write-side materialize gates resolved the connect-aware stream cap by inspecting a live SparkSession, but DataSource V2 write(iterator) is pickled and runs per-partition on workers. On a Serverless worker there is no session, so the cap silently fell back to 256 MiB classic instead of 64 MiB and a 64-256 MiB tile was mis-gated "stream" -> full materialize -> OOM. Fix 1 (HIGH): materialize_decision gains cap_bytes, used directly when given (skips session resolution). RasterGbxWriter and CogGbxWriter capture the cap in __init__ (driver, via SparkSession.getActiveSession()) into self._cap and pass cap_bytes=self._cap in write(). rst_fromfile bakes the driver cap as an f.lit and _fromfile_impl (which runs inside a worker UDF) consumes it via cap_bytes; the read gate file_ref_arg already baked the cap at plan-build (unchanged). Fix 2 (MEDIUM): the "stream" branch decodes the whole raster into RAM, which can be ~10x the compressed on-disk size. RasterGbxWriter.write and rst_fromfile(materialize=True) now gate on the DECODED size read from the tile header, not the compressed size. The read gate stays on compressed size (it holds only raw bytes, decoding one window). Fix 3 (MEDIUM, doc): large tile + pending warp/clip still falls back to full materialize (accepted limitation; WarpedVRT streaming is backlog). Added one test documenting the fall-back path is taken. Fix 4 (LOW): empirically confirmed _windowed_materialize_bytes and materialize_to_bytes emit identical tiled/blockxsize/blockysize/interleave, so those keys were added to the profile-equivalence assertion (strengthened). Fix 5 (LOW): softened the regression-guard docstring — it is a textual tripwire that the gate call is not deleted, not proof of runtime Serverless safety. Verified: 100 passed across the six affected suites in Docker; python lint clean. Co-authored-by: Isaac
… (parity with read gate) RasterGbxWriter.__init__ and CogGbxWriter.__init__ called bare SparkSession.getActiveSession() to bake the connect-aware materialize cap. In some DBR 14+/Connect threading contexts getActiveSession() returns None, leaving both writers with the 256 MiB classic fallback — a Serverless mis-gate. All read-gate surfaces (file_ref_arg, rst_fromfile) already use _resolve_session_for_cap() (getActiveSession → getOrCreate fallback), so this closes the asymmetry: all four cap-capture sites now resolve the session the same robust way. Co-authored-by: Isaac
Introduces MosaicOptions dataclass + parse_mosaic_options() in cog_writer.py: parse and validate mosaic-mode options before any I/O. - Mosaic mode triggers when `mosaic` is truthy or `gridSystem` is supplied; absent / mosaic=false → unchanged single-COG path. - Phase A: only gridSystem='none' (native tiling) accepted; quadbin/ bng/h3 raise ValueError with a clear, user-facing message. - Contradictory combinations rejected at plan time: driverMode + mosaic, tileSize/overlapPercent + DGGS, gridMin/Max/StepResolution + none. - Per-tile encoding options (compress/blockSize/predictor) pass through. - CogGbxWriter.__init__ gains mosaic_opts= (None = single-COG, no behaviour change); cog.py calls parse_mosaic_options upfront so Tasks 2-3 can read self.mosaic_opts without re-parsing. - 29 pure-Python tests; no regression in 3 existing cog_writer suites. Co-authored-by: Isaac
Implements CogGbxWriter._write_mosaic(): when mosaic_opts is set, each source is opened once and tiled into bounded, non-overlapping mini-COGs via rasterio windowed reads. Each window is memory-bounded by the tile grid (decoded_size <= tile_size^2 × bands × itemsize; block-streaming fallback via _windowed_materialize_bytes for the rare over-cap case). New module-level symbols: _tile_grid_windows(w, h, tile_size, overlap_pct) — row-major Window generator with optional halo expansion and source-bounds clamping. _is_all_nodata(data, nodata) — pruneEmpty helper (NaN-aware). _MOSAIC_DEFAULT_TILE_SIZE = 1024. write() branches on self.mosaic_opts first; single-COG and tile-envelope paths are unchanged. CogCommitMessage.paths carries written mini-COG paths for Task 3 VRT build; pending_paths is always empty in mosaic mode. 21 new tests: grid coverage, pixel fidelity, pruneEmpty, overlapPercent halo, pickle round-trip, non-divisible dims, single-COG regression. Co-authored-by: Isaac
commit() in mosaic mode now calls _build_mosaic_vrt() when write_vrt=True (the default). The helper is pure Python + rasterio (no osgeo/GDAL bindings), compatible with the light tier on Serverless. Implementation details: - Opens each mini-COG member with rasterio to collect transform / dims / CRS / dtype / nodata. - Computes the mosaic bounding box from member extents; derives rasterXSize, rasterYSize, and GeoTransform via round() (float-safe for native-tiled grids). - Emits a VRTDataset XML with one VRTRasterBand per source band, each containing one SimpleSource per tile (SrcRect always full tile, DstRect offset = pixel position within the mosaic). - vrt_paths="relative" (default): relativeToVRT="1" + bare filename -> portable, moveable alongside the tiles. - vrt_paths="absolute": full path + relativeToVRT="0". - write_vrt=False: skips the VRT entirely (mosaic_opts=None single-COG path is unchanged). Tests (test_mosaic_vrt.py, 11 new): VRT created; relative paths by default; width/height/CRS/transform match source; windowed read == source pixels; write_vrt=False; absolute paths; only written (non-pruned) tiles referenced; single-COG commit() regression; multi-band VRTRasterBand count. Co-authored-by: Isaac
Add _mosaic.mint_vrt(tile_paths, out=None) — the Phase A "mint" face of the mosaic API. Builds a GDAL VRT over any dynamic tile set (filtered query result, ad-hoc list) and returns the path to a transient file the caller opens and discards. Design: - Reuses _build_mosaic_vrt (Task 3, pure xml.etree + rasterio) via a tempfile.mkdtemp() holding-dir; vrt_paths="absolute" so member paths resolve regardless of where the transient VRT lives. - out= moves the VRT to a caller-supplied destination (temp dir cleaned up). - No osgeo / Spark session / _jvm — light-tier / Serverless safe. Tests (python/geobrix/test/ds/test_mosaic_mint.py): 6 TDD tests: open-as-mosaic, windowed-read, dynamic-subset, transient-location, out-path, no-osgeo; 11+21 regression tests pass. Co-authored-by: Isaac
Detects a .vrt path in RasterGbxReader.partitions() and expands it into one whole-file virtual tile per member mini-COG instead of treating the VRT as a single opaque source. Member paths are extracted by parsing the VRT XML (SourceFilename elements; relativeToVRT="1" resolved relative to the VRT directory, absolute paths used as-is) using defusedxml (with stdlib fallback). Multi-band VRTs reference each tile once per band; deduplication preserves first-occurrence order. Reuses the existing whole-file virtual-tile emission path entirely — _plan_partitions_for_file with emit_virtual=True, window=None. All existing AOI/split options compose unchanged. CogGbxReader inherits the behaviour for free (no code change in cog.py); both raster_gbx and cog_gbx now expand .vrt, giving the write→read round-trip. Load convention: point at the .vrt explicitly to trigger expansion. Directory walk (pointing at a containing directory) is unchanged and still includes .vrt files as rasterio sources. Connect-safe: member enumeration is pure Python / stdlib I/O with no SparkSession, no _jvm. Tests: test_vrt_read_expand.py (6 tests — count, whole-file-virtual struct, rst_avg unchanged proof, minted-VRT absolute-paths, cog_gbx round-trip, no-osgeo lint); all pass. No regression in test_raster_datasource, test_reader_layout, test_raster_virtual, test_cog_reader, test_mosaic_vrt, test_mosaic_mint, test_mosaic_write. Co-authored-by: Isaac
Task 6 of the Phase-A native mini-COG + VRT mosaic SDD. Adds python/geobrix/test/ds/test_mosaic_roundtrip.py with three end-to-end integration tests that prove Tasks 1–5 compose correctly: 1. test_prepare_expand_rst_avg: cog_gbx mosaic write → raster_gbx VRT expansion → rst_avg. Asserts row count == member count, rst_avg values match direct rasterio means per tile, and member pixel means match the source region pixels (pixel-perfect write). 2. test_windowed_via_mint_vrt: mint_vrt over the mini-COGs → rasterio open → cross-tile windowed read. Asserts pixel-exact equality with the same window from the original source (straddles tile_0_0 / tile_0_1). 3. test_bbox_filtered_vrt_read: mosaic.vrt loaded with clipPolygons inset within tile_0_0's spatial extent → asserts exactly 1 row returned for the correct member. No composition gaps found. All five existing mosaic suites are green. Co-authored-by: Isaac
Fix-round for Phase A native mini-COG + VRT mosaic (light tier, pure Python). FIX 1 (headline, silent data loss): _write_mosaic named tiles tile_<row>_<col>.tif in a flat out_dir with no source discriminator, so two source rasters in one write (or two partitions sharing out_dir) collided -- cog_skip_if_exists skipped one / the other overwrote it, losing pixels and leaving a VRT presenting one source as the whole mosaic. Add _source_discriminator (stable per-source, single alnum token, no underscore) and name tiles tile_<disc>_<row>_<col>.tif. The disc hashes the resolved local path, so the same source is deterministic across partitions (idempotent) while distinct sources never collide. FIX 3 (tiled-source edge tiles): a tiled/COG source profile carries tiled=True + blockxsize/blockysize (e.g. 512); a partial-edge tile smaller than the block carried those into the intermediate GTiff write. Drop tiled/blockxsize/blockysize/interleave before writing the window (the final COG re-tiles regardless) -- correctness-hardening. FIX 4 (overwrite skips stale VRT): overwrite cleanup globbed only *.<ext>, stranding a stale mosaic.vrt. Also sweep *.vrt. FIX 5 (dir-load double-counts VRT + members): the raster directory walk read mosaic.vrt as one whole-mosaic source on top of the tile_*.tif it indexes. Exclude *.vrt from _list_source_files (a .vrt is honored only when the load path points directly at it). Tests: two-source no-collision (+VRT references both), tiled-source partial-edge tiles, overwrite removes stale vrt, dir-load excludes vrt, plus updated discriminator-dependent naming assertions. All green in Docker (gbx:test:python): 111 passed. Co-authored-by: Isaac
_parse_vrt_members prefers defusedxml but falls back to stdlib xml.etree when it is absent; defusedxml was in no extra, so a production light install parsed untrusted external .vrt indexes with the vulnerable stdlib parser (XXE / billion-laughs). Add defusedxml>=0.7,<1 to [light-base] so [light], [light_dbr19], and the [*_all] umbrellas inherit it; keep the import fallback for resilience. Lock note: uv is present in the dev container but has no PyPI network, so a full `uv pip compile` regeneration is not possible here. defusedxml 0.7.1 is dependency-free and pure Python, so the only correct change to each hash-pinned lock is adding its two-hash block -- inserted by hand using the repo-trusted hashes already checked in via requirements-dev-container.txt. The two flat .in files list it explicitly; the two -all .in files (.[light_all]/.[light_dbr19_all]) pick it up from the [light-base] change on any future recompile. Follow-up (non-blocking): re-run uv pip compile in a network-enabled environment to confirm byte-parity (only expected delta is the defusedxml block). Co-authored-by: Isaac
…Databricks Spatial Page previously opened raster-only and mentioned vector only as an afterthought. Rewrites the intro, the connect-aware cap section, the FILE fast path section, and the Serverless-safe tables to cover raster and vector consistently throughout. Adds a dedicated "Vector memory profile" section documenting bbox/where pushdown, GPKG/FileGDB worker-local staging, and the FILE-column vector table fast path — all sourced from pyvx/file_read.py and ds/vector.py. Clarifies that the connect-aware cap governs raster tile materialization only; vector's memory is bounded by pushdown + FILE table, not by that threshold. Corrects the writer note: driver commit streams one Arrow batch at a time (not chunkSize-bounded, which is a reader-side option); GeoJSONL has no driver merge at all; OpenFileGDB requires native GDAL / classic clusters. sidebars.js moves the page to right after 'databricks-spatial'. Co-authored-by: Isaac
…e-rasters/virtual-tiles New page docs/docs/api/vrt-mosaic.mdx covers: - Write: cog_gbx mosaic mode options (gridSystem, tileSize, overlapPercent, mergeStrategy, pruneEmpty, writeVrt, vrtPaths) with example - mint_vrt: on-demand transient VRT API (parameters, returns, Connect-safe note) - Read: raster_gbx/cog_gbx .vrt expansion → one virtual tile per member, directory vs VRT load distinction, clipPolygons spatial filtering - Serverless note: windowed tile reads are bounded; source never fully in RAM - Upcoming: grid-aligned mosaics (quadbin/BNG/H3) + windowed rendering Cross-links added to: readers-writers (Next Steps), writers/cog (new VRT mosaic section + Next steps), readers/cog (Next steps), api/large-rasters (See also — 308 MB → 130-tile proof), api/virtual-tiles (See also). sidebars.js: api/vrt-mosaic inserted after api/virtual-tiles in RasterX group. Co-authored-by: Isaac
The Large Rasters "Memory footprint / standard Serverless" section discusses the ~1 GB per-task ceiling but didn't point at the authoritative Serverless & Memory page. Add an inline cross-link to the connect-aware memory model (stream cap, materialize-vs-virtual, FILE Delta-table fast path). Docs build clean. Co-authored-by: Isaac
…ded vs DBLabs Mosaic)
The public option key "mosaic" on the cog_gbx writer collided with the
DBLabs Mosaic project that GeoBrix succeeds. .option("mosaic", true) reads
as invoking the old project. Rename to "vrtMosaic" — unambiguous, matches
the VRT-mosaic feature name — across the parser, all five test files, and the
vrt-mosaic / cog writer doc pages. Beta has no aliases; old key is gone.
Co-authored-by: Isaac
The opening sentence framed a VRT mosaic as strictly a directory of bounded mini-COG tiles. A VRT indexes whole COGs just as well as the mini-COGs the writer produces; broaden the definition so users don't read the mini-COG tiling as a precondition of the mosaic form. Co-authored-by: Isaac
…stem-required The COG writer page buried VRT-mosaic capability in a late section and still carried a pre-rename `mosaic=true` in its prose; the reader page only linked to VRT Mosaics without explaining that cog_gbx expands a .vrt. Add a top-of-page VRT summary to both, a writer mosaic-mode options table, and a reader "Reading a VRT mosaic" section spelling out the options that drive a VRT load (.vrt recognition, clipPolygons+clipCrs filtering, virtualTiles). Also correct a misleading example: gridSystem defaults to "none" (parse_mosaic_options, cog_writer.py:481) and mosaic mode is triggered by vrtMosaic alone, so gridSystem is not required. Drop the redundant gridSystem="none" option from the examples on both the VRT Mosaics and COG writer pages and fix the option-table default from '—' to 'none'. Co-authored-by: Isaac
…lume) The mint_vrt example used out=/tmp/query_mosaic.vrt without explaining that /tmp is driver-local and ephemeral, which read as a reachable persist target. mint_vrt bakes ABSOLUTE member paths (_mosaic.py:88, vrt_paths=absolute), so usability depends on both the .vrt file and its member tiles resolving wherever it is opened. Show both a driver-local (/tmp, transient one-shot read) and a /Volumes (shared, durable, worker/external-client readable) example, add a reachability note, and point users to the writer's relative-path mosaic.vrt when they want a portable movable index instead. Co-authored-by: Isaac
A mosaic.vrt need not originate from the cog_gbx writer — mint_vrt or any GDAL tool can produce one. Loosen "written by" to "can be written by" so the writer reads as one source of a VRT, not the only one. Co-authored-by: Isaac
The VRT Mosaics page opened with prose only. Add a house-style left-to-right hero diagram so the shipped native pattern is graspable at a glance and the image is reusable in slides. The generator emits hand-built SVG (Inter typography, shared stage-card chrome, connector arrows), rasterized to PNG via Chrome headless at 2x and cropped with PIL, matching the sibling rasterx diagrams. Four stages: Source (one large raster too big for a single COG or per-task RAM) -> Tile (the cog_gbx writer in mosaic mode reads window-by-window into bounded mini-COGs) -> Index (the portable mosaic.vrt lens, persisted or minted on demand, openable in any GDAL tool) -> Read (a reader expands it into one virtual tile per member for distributed per-tile rst_* work). Reserved teal marks only the expanded virtual member tiles; mini-COG files and stage chrome use the neutral accents. Embedded at the top of the page with full-sentence descriptive alt text. Co-authored-by: Isaac
serverless-and-memory.mdx sits at the docs/docs/ top level, so its two ../readers-writers links resolved above /docs/ to a 404. Its sibling links already use ./api/... correctly; make the readers-writers links ./readers-writers to match. onBrokenLinks='warn' let these slip past the build (see loose-ends note re: the build script's misleading success). Co-authored-by: Isaac
onBrokenLinks='warn' means docusaurus build exits 0 even with broken internal links, so the script's flat '✅ links resolve' banner was misleading (it masked two real 404s on serverless-and-memory earlier). Add --strict: capture build output, detect Docusaurus's broken-link report, and exit non-zero when present — a green validation gate for pre-push/CI. Default stays permissive (warn, exit 0) so the routine build + dev-server-restart flow isn't blocked by a stray link. Under --strict the dev server is still restarted (port 3000 never left down); only the exit code fails. The success banner now claims 'all links resolve' only when Docusaurus reported none broken; otherwise it warns. Co-authored-by: Isaac
mint_vrt's default temp-dir VRT was caller-cleanup. Add minted_vrt, a contextmanager wrapping mint_vrt that removes the temp dir (and the .vrt in it) on exit, including on exception. Member tiles are absolute-path references, never copied, so cleanup never touches them. mint_vrt itself is unchanged (additive). +2 tests (happy path + exception path); 8 passed. Co-authored-by: Isaac
_build_mosaic_vrt applied the single dataset-level ds.nodata to every VRT band. Switch to rasterio ds.nodatavals (one entry per band) so differing per-band nodata is carried into the VRT; behavior-identical for the common uniform case. Unknown dtype already falls back to Float32 (unchanged). Adds a guard test (multiband uniform nodata gives NoDataValue on every band). Co-authored-by: Isaac
The VRT Mosaics page showed three inline Python blocks (prepare, read/expand, mint) that nothing executed, violating the repo rule that tests are the documentation source. Move the code into vrt_mosaic_examples.py as display constants (VRT_PREPARE / VRT_READ_EXPAND / VRT_MINT) plus executable functions that run the real cog_gbx mosaic writer, raster_gbx VRT expansion, and mint_vrt windowed read against sample data with assertions. The page now renders them via CodeFromTest so the shown code is the tested code. Inline out= fragments and the spatial-filtering snippet stay inline as illustrative fragments. Co-authored-by: Isaac
mjohns-databricks
requested a deployment
to
runtime
August 23, 2026 01:33 — with
GitHub Actions
In progress
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
GeoBrix 0.5.0 — release PR (
beta/0.5.0→main)Promotes the 0.5.0 line to
main(~496 commits). Versions are bumped (pom.xml+__version__=0.5.0). Full change list:docs/docs/release-notes.mdx→ "What's new in v0.5.0".Highlights
cog_gbx/raster_gbx/gtiff_gbxreaders can emit virtual tiles (sourcepath+ pixelwindow, no bytes); pixels are read lazily one window at a time. A virtual row is ~100 B vs 148–527 KB materialized (~1,400–5,000× smaller), dissolving Serverless OOM on large-raster fan-out. Lightweight raster readers now default to virtual (breaking behavior change).cellid, raster (nullable), path, window, clip_polygon, clip_crs, crs, metadata. Both tiers read v1 and v2 and always emit v2.rst_*is virtual-tile-aware via one shared open path (header-only accessors; reference/passthrough ops stay virtual; pixel ops materialize only their window). New force-output params:virtualize_dir/virtualize_prefix/materialize.file_gbx(path lister) +cog_gbxwriter (master-COG prep) +cog_gbxreader (COG-aware windowed read:tileSize/overlapPercent/clipPolygons/windows). ReadersplitStrategynow defaults tonone.gbx_st_{crs,setcrs,transformcrs}(VectorX) andgbx_rst_{crs,setcrs,transformcrs}(RasterX): authority-string CRS handling that survives non-EPSG round-trips.DESCRIBE FUNCTIONsignatures, and many fixes.Pre-merge gate (requested)
docs/docs/notebooks/eo-series.mdx) and the eo-seriesREADME.mdreflect the support, validated on Serverless.Opened as draft until the gate above is met.
beta/0.5.0CI is the dev-branch signal (not a merge gate); themainbuild runs on merge.This pull request and its description were written by Isaac.